Skip to content

feat(skills): quality gates, project/global scope boundary, LLM curation - #120

Merged
jkyberneees merged 10 commits into
mainfrom
feat/skills-quality-gates
Jul 27, 2026
Merged

feat(skills): quality gates, project/global scope boundary, LLM curation#120
jkyberneees merged 10 commits into
mainfrom
feat/skills-quality-gates

Conversation

@jkyberneees

Copy link
Copy Markdown
Contributor

Summary

A hardening pass over the skill-learning pipeline with two goals: stop creating garbage skills, and project-related skills must never be promoted to global skills.

Project → global boundary

  • Scope-gate split (selfimprove.go): NonReusableReason (machine-specific: absolute home paths) drops suggestions entirely; ProjectScopeReason (project-specific: repo-rooted ./scripts/... invocations, hardcoded release version tags) redirects them to ./.odek/skills instead of the global dir. Redirect is refused when the project dir resolves to the global one (odek run from $HOME).
  • Curator confinement (curator.go): MicroCuration / RunAutoCurate / ExecuteMicroCuration are confined via Skill.Source.Dir to skills physically stored under the global dir — previously a project skill overlapping a global draft would be merged into a global skill, silently promoting project content (fail-closed on empty source).
  • Scope UX: skill_save accepts scope: "project"; the interactive learn prompt has a p option. Both refuse when no distinct project dir exists.
  • Redirected skills stay inert until odek skill promote (project-dir skills are forced NeedsReview on scan, as before).

Anti-garbage gates

  • DetectCorrection tightening: fires only on explicit correction phrases and requires the corrected commands to share a lead verb (shareLeadVerb) — kills the corrected-cd garbage class.
  • Keyword hygiene: trigger-keyword derivation filters shell operators, flags, and version fragments; extractTopic skips shell plumbing (cd, sudo, operators, paths).
  • Recurrence gate (candidates.go): a pattern must recur in auto_save.min_occurrences distinct sessions (default 2, 1 disables) before auto-save; first sightings are tracked in ~/.odek/skills/.candidates.json (30-day pruning) and reported as pending. ⚠️ Behavior change: first-time patterns no longer save immediately.
  • Score ranking: eligible suggestions are sorted by a deterministic substance score so max_per_run keeps the strongest.
  • Near-duplicate rejection: stopword-filtered Jaccard word-set similarity ≥ 0.85 against an existing skill in the target dir skips the save; same-name saves count as updates.
  • Secret scan: WriteSkill (single choke point for all save paths) runs internal/redact over body + description — matches become [REDACTED] and the skill is pinned to NeedsReview.

LLM-enhanced curation

With llm_curate enabled, merge bodies are synthesized by the model (structure/size sanity-checked, mechanical concatenation as fallback) instead of accumulating --- Merged from X --- sections. The worst-of provenance union is preserved either way, so a merge can never launder taint.

Also in this PR

  • test(maintenance): TestStartRunsSweepOnTick no longer waits for the real 1-minute janitor tick (63s → ~1s per run; it was blowing past 60s test bounds and looking like a go test hang).
  • fix(config): the field-wise auto_save merge now honors min_occurrences (was silently dropped).

Test plan

  • go test ./... -count=1 — all 29 packages ok
  • go test -race ./internal/skills/ ./internal/config/ ./cmd/odek/ -count=1 — ok
  • New tests: scope-gate classifiers, redirect + edge cases (projectDir == userDir), curator confinement regression, recurrence gate (pending → save on second session, pruning, fingerprint stability), score ranking, near-dup (skip/distinct/update), LLM merge (synthesized + both fallbacks), skill_save scope (project/same-dir refusal/invalid), WriteSkill redaction (leaky + clean)
  • Docs synced: LEARNING.md, CONFIG.md, SECURITY.md, AGENTS.md

TestStartRunsSweepOnTick waited for the janitor's real 1-minute tick,
making internal/maintenance take 63s on every run and blowing past any
60s test bound (it looked like a hang under 'go test'/'go test -race').
Start now honors a package-level tickInterval test hook; the test
overrides it to 25ms. Production behavior is unchanged.
Anti-garbage:
- derive.go: filter shell operators, flags, and version fragments from
  derived trigger keywords.
- DetectCorrection: fire only on explicit correction phrases and require
  the corrected commands to share a lead verb (shareLeadVerb), killing
  the 'corrected-cd' garbage class.
- extractTopic/leadVerb: skip shell plumbing (cd, sudo, operators,
  paths, flags) so topics resolve to real verbs.

Project -> global boundary:
- Split the reusability gate: NonReusableReason (machine-specific:
  absolute home paths) drops suggestions entirely; ProjectScopeReason
  (project-specific: repo-rooted ./scripts/... invocations, hardcoded
  release version tags) redirects them to ./.odek/skills instead of the
  global dir. Redirect is refused when the project dir resolves to the
  same location as the global dir (odek run from $HOME).
- Confine micro-curation (MicroCuration/RunAutoCurate) to skills
  physically stored under the global dir via Skill.Source.Dir, so a
  project skill can never be merged into a global skill or deleted by
  the curator (fail-closed on empty source).
- Wire projectDir through RunAutoSaveLoop and the CLI/serve/telegram
  call sites; reload + notifier events account for ProjectSaved.

Docs: LEARNING.md design decisions, AGENTS.md security architecture.
Skill bodies are session content written to disk permanently; a command
captured with credentials in it (curl -H 'Authorization: ...', export
AWS_..., etc.) would be immortalized in ~/.odek/skills. WriteSkill now
runs internal/redact over body + description: matches are replaced with
[REDACTED] and the skill is pinned to NeedsReview so the operator sees
why before it can auto-load. Covers every write path (auto-save,
interactive prompt, skill_save, curator merges, imports).
AutoSaveSuggestions kept the first MaxPerRun eligible suggestions in
heuristic emission order, so weak patterns could crowd out strong ones.
Eligible suggestions are now sorted by a deterministic substance score
(verification section, command-log evidence, body length, description,
LLM-judged provenance) before the cap is applied.
A single quirky session could previously produce a saved skill. The
auto-save pipeline now records each suggestion's fingerprint
(heuristic+name, stable across sessions) in ~/.odek/skills/.candidates.json
and only saves once the pattern recurs in a later session
(auto_save.min_occurrences, default 2; set 1 to disable). Pending
suggestions are reported in AutoSaveResult.Pending and verbose output.
Candidates unseen for 30 days are pruned.
Dedup was exact BodyHash only (and post-hoc via curation), so light
paraphrases accumulated as separate skills. AutoSaveSuggestions now
compares each quality-gated suggestion against existing skills in the
target dir with stopword-filtered Jaccard word-set similarity (>= 0.85
= duplicate, recorded in AutoSaveResult.DuplicateOf). Same-named saves
bypass the gate: re-saving a name is an update, not a duplicate.
Mechanical merges concatenated bodies with a 'Merged from X' separator,
producing ever-growing, contradictory skill files. When llm_curate is
enabled and an LLM client is available, ExecuteMicroCurationWithLLM now
asks the model to synthesize one deduplicated body (sanity-checked for
structure and size); any failure or invalid output falls back to the
mechanical merge. The worst-of provenance union from MergeSkills is
kept either way, so an LLM merge can never launder taint.
…ctive prompt

The project/global boundary only existed in the unattended pipeline.
skill_save now accepts scope: "project" to save project-specific
procedures to ./.odek/skills (refused when no distinct project dir
exists, e.g. odek run from $HOME), and the interactive learn prompt
gained a 'p' response with the same semantics. Both paths keep the
existing NeedsReview pinning.
The field-wise AutoSave merge in internal/config silently dropped the
new min_occurrences key. Also pins min_occurrences=1 in
TestRunLearn_MultiStepProcedure (it covers the save path; the gate has
its own unit tests) and documents the new gates in LEARNING.md,
CONFIG.md, and AGENTS.md.
… new gates

Interactive prompt examples now show the p=project option, config
snippets include auto_save.min_occurrences, the LEARNING.md limitations
section reflects save-time dedup, and SECURITY.md's skill provenance
section documents the scope gates and save-time hygiene (recurrence,
near-dup rejection, secret redaction).
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
odek 6bb2ea1 Commit Preview URL

Branch Preview URL
Jul 27 2026, 05:52 PM

@jkyberneees
jkyberneees merged commit f76d5f2 into main Jul 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant